Method and apparatus for locating candidate data centers for application migration

ABSTRACT

When an application running at a first data center needs to be migrated to a different data center, the first data center is able to automatically locate an appropriate target data center from among multiple other data centers for receiving the migration of the application. The first data center ensures that a specified level of performance of the application will be maintained following migration of the application by preparing a migration request document that specifies network requirements, server requirements and storage requirements for the application. A targeted data center that is a candidate for receiving migration of the application receives the specified requirements and uses a performance index and other analysis to determine whether its hardware, software and other configurations are capable of meeting the specified requirements. The results of the determination are sent back to the requesting data center, which then decides if migration will be carried out.

CROSS-REFERENCES

This is a continuation application of U.S. No. 11/987,284, filed Nov.29, 2007, the entire disclosure of which is here by incorporated byreference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates generally to information systems and datacenters.

2. Description of Related Art

Large companies and other enterprises may have multiple informationsystems and data centers that they use to conduct their business. Forexample, carriers who provide phone and Internet-related services willgenerally have multiple geographically-dispersed data centers to covertheir service area. These enterprises may be running a variety ofdifferent services including voice transmission, e-mail, Internetaccess, messaging, video streaming, and the like, using networks,servers and storage systems provided by the data centers. Thus, theeffective and efficient use of resources such as the networks, serversand storage systems in these data centers is necessary for thesuccessful operation of these enterprises.

Recently, virtualization of networks, servers and storage has increasedthe efficiency with which data centers can be operated. For example,server virtualization is a technology that enables server consolidationin certain information system arrangements, such as at data centers, byallowing single physical servers to provide a plurality of virtualserver environments using virtual machine software. Under thistechnology, one or more physical servers can be divided into multiplevirtual server environments instead of having multiple dedicatedphysical servers, each running a different server environment. Further,virtual machine software is also able to combine several physicalservers to create a single virtual server. Server virtualization can beused to eliminate the requirement for having a large number of differentphysical servers in a data center, and thereby enable more efficient useof server resources, while improving server availability. Also, servervirtualization can help reduce overall costs, reduce power consumption,reduce time for server provisioning, centralize server management andadministration, assist in agile service deployment and improve disasterrecovery capabilities.

Virtualization technology allows the data center management to changethe system configuration dynamically in order to improve performance ofapplications running in a data center. Furthermore, migration of anapplication between data centers that are distributed over a wide areacan be carried out by forwarding the data which is necessary for runningan application. However, problems currently exist for resourceprovisioning to enable application migration between environmentsincluding geographically-dispersed data centers. For example, it isdifficult to find an appropriate target data center to which to migratean application while maintaining the same performance as before themigration because applications typically utilize multiple layers ofdifferent technologies including networks, servers and storages. Inparticular, network devices, server devices and storage devices whichare used to operate an application may be different at every datacenter, and determining available capacity of these devices is notalways possible. Therefore, the application may not operate at the samelevel of performance when operation of the application is moved from afirst data center to a different data center. In addition, even when anapplication can be migrated to the other data center and maintain thesame level of performance, the migration may affect the performance ofother applications running in the targeted data center. Existing methodsare just able to carry out a configuration change within the data centerfor performance improvement after the applications are migrated.

Related art includes U.S. Pat. No. 7,099,912, to Ishizaki et al.,entitled “Integrated Service management system”; U.S. Pat. No.7,062,559, to Yoshimura et al., entitled “Computer Resource AllocatingMethod”; and U.S. patent application Ser. No. 11/892,045, to Oeda, filedAug. 20, 2007, entitled “Storage and Service Provisioning forVirtualized and Geographically Dispersed Data Centers”, the entiredisclosures of which are incorporated herein by reference.

BRIEF SUMMARY OF THE INVENTION

The invention offers a solution for automatically locating one or moreappropriate data centers that are able to run an application accordingto a current state of performance to enable migration of theapplication. Embodiments of the invention provide for dynamicallychanging resource allocation in a data center according to a load on thedata center, and maintaining performance and security for each user whenthe load of a user fluctuates. These and other features and advantagesof the present invention will become apparent to those of ordinary skillin the art in view of the following detailed description of thepreferred embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, in conjunction with the general descriptiongiven above, and the detailed description of the preferred embodimentsgiven below, serve to illustrate and explain the principles of thepreferred embodiments of the best mode of the invention presentlycontemplated.

FIG. 1 illustrates an example of a hardware configuration andrelationship between multiple data centers in which the method andapparatus of the invention may be applied.

FIG. 2 illustrates an example of a hardware and software configurationof an individual data center.

FIG. 3 illustrates an example of a network management server.

FIG. 4 illustrates an example of a server management server.

FIG. 5 illustrates an example of a storage management server.

FIG. 6 illustrates an exemplary data structure of a VPN resource table.

FIG. 7 illustrates an exemplary data structure of a virtual machineresource table.

FIG. 8 illustrates an exemplary data structure of a CPU process timetranslation table.

FIG. 9 illustrates an exemplary data structure of a storage volumeresource table.

FIG. 10 illustrates an exemplary data structure of a federated datacenter table.

FIG. 11A illustrates an exemplary data structure of a migration requestreceived table.

FIG. 11B illustrates an exemplary data structure of a migration requestsent table.

FIG. 12 illustrates an exemplary data structure of a migration requestdocument.

FIG. 13 illustrates an exemplary process flow of a process for locatingan appropriate data center according to the invention.

FIG. 14 illustrates an exemplary process flow for checking availablebandwidth.

FIG. 15 illustrates an exemplary process flow for server availablecapacity confirmation.

FIG. 16 illustrates an exemplary process flow for storage availablecapacity and performance confirmation.

DETAILED DESCRIPTION OF THE INVENTION

In the following detailed description of the invention, reference ismade to the accompanying drawings which form a part of the disclosure,and, in which are shown by way of illustration, and not of limitation,specific embodiments by which the invention may be practiced. In thedrawings, like numerals describe substantially similar componentsthroughout the several views. Further, the drawings, the foregoingdiscussion, and following description are exemplary and explanatoryonly, and are not intended to limit the scope of the invention or thisapplication in any manner.

Appropriate networks, servers and storage systems are necessary forrunning an application at a proper level of performance. In order tomaintain the performance of the application at a certain level, theperformance of each of these elements needs to be monitored andmaintained. Embodiments of the invention disclose a system and methodthat confirms whether a data center that is a candidate for receivingmigration of an application will be able to maintain the performance ofthe application if the application is migrated. The invention canachieve this by checking that the data center is able to providesufficient network bandwidth, sufficient server computing power andadequate storage capacity and access speed from a suitable server.

Embodiments of the invention include an integrated management systemthat determines whether a certain application can be migrated to anotherdata center by communicating with the integrated management systems ofother data centers. The integrated management system generates amigration request document that contains network requirements, serverrequirements and storage requirements. The migration request document issent by the originating data center to the other data centers. Theintegrated management systems of the data centers that receive themigration request documents check the situation of their localenvironment. Each integrated management system checks the network layer,the server layer and the storage layer. Regarding a network layer, theintegrated management system determines whether there is sufficientavailable bandwidth to meet the specified requirement. Regarding theserver layer, the integrated management system determines whether thereis a specified virtual machine manager and sufficient available servercapacity to meet the specified requirement. Regarding the storage layer,the integrated management system determines whether there are availablevolumes having free capacity to meet the specified requirement. Theintegrated management system checks whether the volumes that theapplication needs to use can be carved from an array group having asimilar access performance pattern without having a detrimentalinfluence on other applications that are running at the data center.

Relationships between Multiple Data Centers

As illustrated in FIG. 1, multiple data centers 10 a, 10 b, 10 c areconnected to each other for communication through a network 12. Datacenters 10 are typically geographically dispersed from each other, andnetwork 12 may be any network type, including WAN (Wide Area Network),LAN (Local Area Network) or the like. Each data center 10 may comprisean integrated management system 111, a network management server 131, aserver management server 132, a storage management server 133, one ormore network switches 141, servers 142 for running one or moreapplications, a SAN switch 143 and storage 144. Network managementserver 131 manages operation of network switches 141 for communicationwith clients. Server management server 132 manages operation andconfiguration of servers 142. Storage management server 133 managesoperation and configuration of SAN switch 143 and storage 144.

Integrated management system 111 includes a migration acceptor program121, a resource manager 122 and migration requester 123. When migrationof an application is to be carried out, a migration request document 13is sent by migration requester 123 of one of data centers 10 a, 10 b or10 c, and the migration request document 13 is received by migrationacceptors 121 of others of data centers 10 a, 10 b, 10 c. Migrationrequest document 13 is processed by resource managers 122 of thereceiving data centers. Further, while three data centers 10 areillustrated in FIG. 1, it should be noted that any number of datacenters may embodied under the invention.

Hardware Architecture and Software Modules of Data Center

FIG. 2 illustrates the details of the hardware architecture and softwaremodules that may be contained in each data center 10. As illustrated inFIG. 2, a management LAN 202 connects integrated management system 111,network management server 131, server management server 132, storagemanagement server 133, a VPN (Virtual Private Network) router/switch 141a, a VLAN (Virtual Local Area Network) switch 141 b, servers 142, SANswitch 143 and storage 144 to enable communication therewith, and toalso enable communication with network 12 via a router 21. Integratedmanagement system 111, network management server 131, server managementserver 132 and storage management server 133 are generic computers thateach include a CPU 222, a memory 223, a NIC (Network Interface Card) 221and a disk drive 224, which are internally connected through a systembus 212. Integrated management system 111 includes migration acceptorprogram 121, migration requester program 123 and resource managerprogram 122, which are loaded in memory 223 a or other computer readablemedium for execution by CPU 222 a. A federated data center table 213, amigration request received table 214, and a migration request sent table215 are stored in the disk storage 224 a, and may also be loaded intomemory 223 a. Integrated management system 111 includes a user interface211 that is used for carrying out functions such as sending a migrationrequest or displaying managed data of the data center to anadministrator, or the like.

Network switches 141 may include VPN router 141 a that enablesrealization of a virtual private network (VPN) 203. Virtual privatenetwork service provides a secure network path between a customer site204 and the data center 10 of the service provider over a shared publicnetwork such as the Internet. Each VPN service customer has one or moresites to be connected to the VPN. One end point of a VPN path is a VPNrouter 205 at a customer's site, and the other end point of the VPN pathis on the VPN router 141 a, which resides in the data center 10. VPNrouter 141 a in the data center 10 aggregates multiple VPN paths,identifies the customer by VPN ID, and directs the traffic received fromthe customer to the virtual LAN VLAN switch 141 b.

Within the VPN router 141 a, multiple virtual router (VR) programs 206may be running, each serving different VPN customers. For example, inFIG. 2, the VPN paths from customer site A 204 a are terminated at thevirtual router A 206 a in VPN router 141 a. Similarly, the VPN pathsfrom customer site B 204 b are connected to the virtual router B 206 b.Because each virtual router 206 has its own routing table and othernetwork resources dedicated to the particular customer it serves,packets from each VPN customer are clearly separated from the networkpoint of view. This allows, for example, two different customers to usethe same overlapping IP address spaces within a private address range.When a virtual router A 206 a receives packets from the VPN path ofcustomer A 204 a, virtual router A 206 a then adds to these packets aVLAN tag for customer A and sends the packets to the VLAN switch 141 b.A VLAN tag is information added to the LAN frame so that more than twologically independent networks are able to be overlaid on the same LANsegment. A more detailed specification and discussion of the user ofVLAN tags is defined in the IEEE 803.1Q standard published by IEEE(Institute of Electrical and Electronics Engineers, Inc.), which isincorporated herein by reference. The technology to manage the qualityof service is, for example, MPLS (Multi Protocol Label Switching)DiffServ (Differentiated Services) which is defined by the IETF(Internet Engineering Task Force).

Servers 142 include one or more physical servers 232 and a serviceprocessor (SVP) 231. In the embodiment illustrated, there are eightphysical servers 232, labeled PS-001 to PS-008, although the number ofphysical servers 232 can vary. Multiple virtual machine managers (VMMs)233 are available and, in embodiments of the invention, customers canchoose one of these on which to run their applications. In FIG. 2,virtual machine managers VMM-001 233 a, VMM-002 233 b and VMM-003 233 care shown as being available. The roles carried out by virtual machinemanagers 233 include combining plural physical servers into one virtualserver and dividing one physical server into plural virtual servers. Forexample in FIG. 2, VMM-001 combines four physical servers PS-001 toPS-004 and then divides the resources of these four physical serversinto two virtual servers (virtual machines) VM-101 234 a and VM-102 234b. Also, VMM-002 233 b combines physical servers PS-005, PS-006, and hasone virtual server VM-201 234 c active thereon, while VMM-003 233 c isonly active on physical server PS-007. On the other hand, physicalserver PS-008 is a raw physical server that does not have virtualizationapplied thereto.

The virtual machine managers 233 are able to assign the CPU power andmemory of the server(s) they control according to the customer needs.For example, each virtual or physical server belongs to a certain VLAN.However, as VLAN switch 141 b on the VLAN segment is configured totransfer VLAN-tagged packets from VPN router 141 a to an appropriateserver connected to the VLAN segment, it is not necessary for servers toknow which VLAN they belong to. Further, each virtual or physical serverhas one or more host bus adapters (HBAs) (not shown) that are connectedto the storage area network (SAN), i.e., to SAN switch 143. From the SANpoint of view, each virtual or physical server is identified by its HBAaddress or port. In the configuration of FIG. 2, each virtual orphysical server is connected to the SAN switch 143.

Storage 144 includes one or more storage systems providing multiplelogical storage volumes 243 for use by servers 142. Each volume 243 maybe carved from an array group 242 that may be composed as a RAID(Redundant Array of Independent Disks). In some embodiments, each arraygroup 242 can be composed by different types of disks and/or differentlevels of RAID, thereby providing different levels of performance. InFIG. 2, volumes Vol-101 243 a and Vol-102 243 b are carved from arraygroup 242 a, volumes Vol-201 243 c and Vol-202 243 d are carved fromarray group 242 b, and volumes Vol-301 243 e and Vol-302 243 f arecarved from array group 242 c. Each array group 242 is controlled bystorage controller 241. Virtual servers 234, physical servers 232 andstorage volumes 243 may be connected via a storage area network (SAN),which may be Fibre Channel, or other type of connection. In FIG. 2, SANswitch 143 is used to connect servers 232, 234 and volumes 243 as a SAN.To segregate servers and volumes for different customers, SAN switch 143and/or storage volumes 242 may be configured so that access to aspecific storage volume is restricted to a specific virtual server 234or physical server 232. For SAN switches, this security functionality iscalled “port zoning”, while for storage devices, this securityfunctionality is often called “LUN security”, where “LUN” stands forlogical unit number. From the performance point of view, volumes whichhave similar access patterns are desired to be on the same array group.The word “access pattern” here refers to a read/write access ratio and arandom/sequential access ratio.

Typically, an application can be run on its own virtual server 234. Forexample, a first application might be run on VM-101 234 a and a secondapplication might be run on VM-102 234 b, and so forth. The firstapplication might use virtual router A 206 a and the second applicationmight use virtual router B 206 b. Further, the first application mightuse volumes Vol-101 234 a and Vol-102 243 b, while the secondapplication might use volumes Vol-201 243 c and Vol-202 243 d. Thus,through virtualization, network, server and storage resources can beapportioned to various applications. Embodiments of the inventionautomatically locate suitable candidate data centers to receivemigration of an application, for example, if a data center where theapplication is currently being operated is overloaded, or if migrationof an application otherwise becomes necessary or desirable.

FIG. 3 illustrates additional details of network management server 131.NIC network interface card 221 b of network management server 131 isconnected to the management LAN 202 and is used to send or receivecommand packets to or from integrated management system 111. Networkmanagement server 131 also includes a VPN configuration program 325loaded in memory 223 b, or other computer readable medium, which isexecuted by CPU 222 c. VPN configuration program 325 communicates withVPN router 141 a for configuring VPN router 141 a, including thecreation of virtual routers 206. The information for mapping between VPNpath, VLAN tags associated with the information, and network bandwidthinformation are stored in a VPN resource table 311 in disk 224 b. Thecontents of the VPN resource table 311 are described further below withreference to FIG. 6.

FIG. 4 illustrates additional details of the server management server132. NIC 221 c connects server management server 132 to the managementLAN 202, and is used to send or receive command packets to or fromintegrated management system 111. Server management server 132 includesa virtual machine resource management program 425 in memory 223 c orother computer readable medium, which is executed by CPU 222 c. Virtualmachine resource management program 425 communicates with VLAN switch141 b to configure the VLAN settings. Virtual machine resourcemanagement program 425 also communicates with one or more virtualmachine managers 233 on servers 142 to manage virtual servers 234.Management of virtual servers 234 includes allocating CPU and memoryresources, starting up and shutting down virtual servers 234, andconfiguring network settings. Virtual machine resource managementprogram 425 also communicates with one or more physical servers 232 orSVP service processor 231 to manage any raw physical servers, such asPS-008 in FIG. 2. Management of physical servers includes allocating CPUand memory resources, logical partitioning, starting up and shuttingdown virtual servers or virtual machine managers, and configuringnetwork settings.

Information of the relationships between virtual machine managers 233,virtual servers 234 and physical servers 232 are stored in a virtualmachine resource table 411 in disk 224 c or memory 223 c. Information onthe mapping between virtual servers 234 and the VLAN is also stored invirtual machine resource table 411. The contents of virtual machineresource table 411 are described in additional detail below withreference to FIG. 7. Furthermore, information for translating a CPUperformance index is stored in a CPU process time translation table 412in disk 224 c or memory 223 c. The CPU performance index and contents ofCPU process time translation table 412 are described further below withreference to FIG. 8.

FIG. 5 illustrates additional details of the storage management server133. NIC 221 d connects storage management server 133 to the managementLAN 202, and is used to send or receive command packets to or fromintegrated management system 111. Storage management server 133 alsoincludes a storage volume management program 525 and a volume accessperformance collector program 526 in memory 223 d or other computerreadable medium, which is executed by CPU 222 d. Storage volumemanagement program 525 communicates with SAN switch 143 to configure theSAN setting, such as by configuring SAN switch 143. Storage volumemanagement program 525 also communicates with storage 144 to obtainvolume information on volumes 243. Information of relationships betweendisk array groups 242 and volumes 243, capacity of each volume 243, freespace of each array group 242, mapping between servers 232, 234 andvolumes 242, and volume access time are obtained by storage volumemanagement program 525 and volume access performance collector program526, and are stored in a virtual machine resource table 511 in disk 224d or memory 223 d. The contents of storage volume resource table 511 aredescribed further below with reference to FIG. 7. VPN Resource Table

FIG. 6 illustrates the VPN resource table 311 that may reside in thenetwork management server 131. VPN resource table 311 containsconfiguration information that VPN configuration program 325 uses toconfigure the VPN router 141 a. VPN-ID 601 identifies which VPN virtualrouter (VR) that a site entry 602 corresponds to (e.g., customer site).Address 1 603 and Address 2 604 hold IP address values of each endpointof a VPN path or tunnel, respectively (for example, the IP address ofthe customer site and the IP address of the virtual router,respectively). Bandwidth 605 indicates the network bandwidth value whichis reserved in VPN router 141 a to maintain a desired level of qualityof service of the network connection between Address 1 603 and Address 2604. VLAN-ID 606 is the VLAN tag value assigned to the packets receivedfrom this site over the VPN path.

Virtual Machine Resource Table

FIG. 7 illustrates the virtual machine resource table 411 that mayreside in server management server 132. Virtual machine resource table411 contains the configuration information that virtual machine resourcemanagement program 425 uses to manage servers 142. Virtual machinemanager 701 identifies the platform of virtual machine software beingused which carries out processes for creating, starting and stopping ofvirtual machine virtual servers. For example, VMM-001 711 might be afirst brand of virtual machine management software, VMM-002 712 might bea second brand of virtual machine management software, and VMM-003 714might be at third brand of virtual machine management software. Examplesof virtual machine software currently used for virtual serverapplications include “VMware Server” available from VMware, Inc. of PaloAlto, Calif., “MS Virtual Server” available from Microsoft Corp. ofRedding, Wash., and “XenServer” available from Citrix Systems of Ft.Lauderdale, Fla., although the invention is not limited to anyparticular virtual machine software. Further, if the value of thevirtual machine manager 701 is “NON-VMM”, as indicated at 716, thismeans that no virtualization mechanisms are used (i.e., the server is araw physical server). According to FIG. 7, four virtual servers VM-101,VM-102, VM-103 and VM-104 are running on virtual machine managerVMM-001, two virtual servers VM-201 and VM-202 are running on virtualmachine manager VMM-002, three virtual servers VM-301, VM-302 and VM-303are running on virtual machine manager VMM-003. One server, PS-008, isrunning as a physical server without virtualization.

In virtual machine resource table 411, virtual server entry 702identifies the virtual machines that currently exist in each virtualmachine manager's environment. For example, VMM-001 711 includes fourvirtual machines currently running. Additionally, if some or all ofvirtual server entry 702 is indicated to be “Free”, then this indicatesthe available capacity that a virtual machine manager can use to createadditional virtual servers using available capacity of CPU 705 andmemory 706. Physical server 703 identifies one or more physical serverswhich are used for realizing server virtualization by each virtualmachine manager 701. CPU type 704 is a CPU type of each physical server703. For example, different servers may have different CPUs installed,depending on age, initial purchase specifications, etc. CPU allocation705 and memory allocation 706 indicate how much CPU processing capacityand memory resources are currently allocated to the virtual servers 702,respectively. Performance capacity 707 indicates the capacity forperformance, which is a summed value of CPU allocation 705 of acorresponding virtual server 702. For example, VM-202, which is createdand managed by VMM-002 712 is allotted 15 percent of the capacity of theCPU of physical server PS-005 and 20 percent of the capacity of physicalserver PS-006 for a performance capacity total of 35 percent of a CPU ofCPU type CPU-001. VLAN-ID 708 is the VLAN tag value assigned to thecorresponding virtual server 702. VLAN-ID 708 can be used by theresource manager program 122 to determine which servers are in the sameVLAN, i.e., which servers are running an application. When a first datacenter (i.e., the Migration Requester) identifies requirements of anapplication to be migrated, as discussed below with reference to FIG.13, the data center needs to identify which servers are included forthat application.

CPU Process Time Translation Table

FIG. 8 illustrates CPU process time translation table 412, which mayreside in the server management server 132. This table containsinformation that is used to translate a performance index depending onthe virtual machine manager and CPU type so that an expected level ofperformance can be predicted for various CPU types and virtual machinemanager software types in various servers. Virtual machine manager entry801 identifies the software platform of virtual machine managementsoftware that carries out creating, starting and stopping of virtualservers. If virtual machine manager entry 801 is “NON-VMM”, then thismeans no virtualization mechanisms are being used. CPU type 802 is theCPU type of a physical server. Performance index 803 is a comparativeindex of performance. The performance index values 803 set forth in FIG.8 are representative of performance measured when a certain program(e.g., a benchmark) was run with a certain VMM and a certain CPU type.The performance of CPU types varies with the VMM even though the CPUtype is the same, and the performance of the VMM varies depending on theCPU type. Therefore, it is necessary to translate performance by usingthis table to make sure that a requested performance level can bereserved for an application to be migrated. For example, at entry 811,the CPU type CPU-001 running VMM-001 carried out a certain benchmarktest in a first period of time. At entry 812, CPU type CPU-002 runningthe same VMM-001 required 120 percent as much time to carry out the samebenchmark test as CPU type CPU-001, and therefore has a performanceindex of 120. Similarly, at entry 817, CPU type CPU-001 running VMM-003required only 65 percent as much time to carry out the benchmark test,and therefore has a performance index of 65. Thus, by compiling the CPUprocess time translation table 412, it is possible to estimate whether aphysical server that is a target of migration will have sufficientprocessing capacity to run an application at a desired performancelevel. Further, while in the performance index illustrated, a largerindex number indicates a lower performance relative to a benchmark scoreof a particular CPU type, it would also be possible to use othermeasuring schemes for determining relative CPU performance, such asdetermining ratios of processing power for various CPU types, or thelike.

Storage Volume Resource Table

FIG. 9 illustrates storage volume resource table 411, which may residein the storage management server 133. Storage volume resource table 411contains configuration information that storage volume managementprogram 525 uses to manage SAN switch 143 and storage 144. Array groupentry 901 identifies the array group which is composed of plural storagemedia, such as disk drives. Disk type 902 indicates the kind of storagemedia that makes up each array group. RAID type 903 indicates the RAIDlevel used in configuring each array group 901. Busy rate 904,Read/Write ratio 905 and Random/Sequential ratio 906 are statisticaldata of each array group 901 collected by storage volume managementprogram 525. Busy rate 904 indicates how busy the particular array groupis, for example, the percentage of time the array group is occupiedperforming I/O operations as measured over a period of time (e.g., I/Ooperations per second). Read/write ratio 905 is the ratio of the numberof read access operations out of the total number of access operationsmeasure over a period of time. Random/sequential ratio 906 is the ratioof random access counts out of total access counts, i.e., the number ofaccesses to disk that require random seeks, as opposed to sequentialreads or writes, which can be an indication of high disk fragmentation.Further, it is desirable to have the same access characteristics forapplications in the same array group. Thus, read/write ratio 905 andrandom/sequential ratio 906 can be used as specified criteria fordetermining a suitable recipient for application migration.

Volume entry 907 identifies the logical storage extents (i.e., logicalvolumes 234) that are currently carved from a particular array group.When the value of volume entry 907 is “Free”, this means that theparticular array group has available storage capacity for creating oneor more new volumes. Capacity 908 is the capacity of the volumes listedin volume entry 907, or the available free capacity remaining. Forexample, in FIG. 9 at line 911, volumes VOL-101, VOL-102 and VOL-103 arecarved from array group AG-001, and AG-001 has a free space for a newvolume to be carved. In FIG. 9, volume VOL-101 has a capacity of 1.0terabytes, and the remaining free capacity in array group AG-001 is 2.5terabytes. Server entry 909 lists the virtual server or physical serverthat uses a particular volume identified in volume entry 907. Forexample, in FIG. 9, VOL-101 is able to be accessed by virtual serverVM-101, and VOL-102 is able to be accessed by the physical serverPS-008. From the server point of view, the virtual server VM-101 usesthree volumes, namely VOL-101, VOL-201 and VOL-203. Access time 910 is astatistical data collected by the volume access performance collectorprogram 526, and indicates an average performance value measured overtime for how quickly the server is able to access the data on theparticular volume.

Federated Data Center Table

FIG. 10 illustrates the federated data center table 213 that may residein the integrated management system 111. The federated data center table213 contains the data center information that the migration requesterprogram 122 uses when making migration requests for determining to whichdata centers the migration requests should be sent. Data center ID 1001is the identifier of each data center and request destination address1002 is the destination address of the data center that is the target ofthe migration request. As discussed below, the migration requesterprogram 122 sends a migration request document 13 to the requestdestination address 1002 for a data center listed in the federated datacenter table 213. The request is received by the migration acceptorprogram 121 at the target data center, and the migration acceptorprogram 121 prepares and responds with a suitable reply. Requestpriority entry 1003 set forth preference information to be used when themigration requester program makes 122 prepares a migration request, andis typically assigned by an administrator. For example, a data centerhaving a higher priority would receive a migration request before otherdata centers having lower priorities.

Migration Request Received and Sent Tables

FIG. 11A illustrates the migration request received table 214 that mayreside in the integrated management system 111. The migration requestreceived table 214 contains the historical records of migration requests(i.e., migration request documents 13) that the migration acceptorprogram 121 has received at the data center maintaining the migrationrequest table 214, and the results generated. ID 1101 is the identifierassigned to the migration request received (MRR). Requester origin entry1102 identifies the data center that the migration acceptor program 121received the request from. Date request was received 1103 records thedate and time that the migration acceptor program 121 received therequest from the other data center. Network result 1104, server result1105 and storage result 1106 show the results of checking the capacityof the network, servers and storage in response to the migrationrequest, as is discussed further below. Migration request contents 1107includes the entire data of the migration request document 13 that themigration acceptor program 121 received.

FIG. 11B illustrates the migration request sent table 215 that may bemaintained in the integrated management system 111 of each data center.The migration request table 215 is similar to migration request receivedtable 214, discussed above, but contains the historical records ofmigration requests (i.e., migration request documents 13) that themigration requester program 123 has sent to other data centers. ID 1121is the identifier assigned to the migration request sent (MRS). Requesttarget 1122 identifies the data center that the migration requesterprogram 123 sent the request to. Date request was sent 1123 records thedate and time that the migration requester program 123 sent the requestto the other data center. Network result 1124, server result 1125 andstorage result 1126 show the results received back regarding whether theother data center was able to meet the requested capacity andperformance for the network, servers and storage in response to themigration request, as is discussed further below. Migration requestcontents 1127 may contain the entire data of the migration requestdocument 13 that the migration requester program 123 sent to the otherdata center.

Migration Request Document

FIG. 12 illustrates the migration request document 13 that migrationrequester program 123 creates and sends to initiate a migration requestto another data center. Migration request document 13 consists of threeparts: a VPN request part 135, a server request part 136 and a storagerequest part 137. VPN request part 135 has information on a requestednetwork, including bandwidth 1211. In FIG. 12, bandwidth entry 1211shows that a bandwidth of 1.5 Mbps is requested. Request ID entry 1212,1231 and 1251 is a unique identifier for this particular migrationrequest document. The request ID serves the role of connecting eachrequest part 135, 136, 137 in the migration request document 13. Serverrequest part 136 contains information on virtual machine manager type1232, CPU type 1233 and CPU allocation 1234 requested. For example, FIG.12 illustrates that VMM-001 is requested as the type 1232 of virtualmachine manager, CPU-002 is requested as the CPU type 1233 and 200percent is requested for CPU allocation 1234 (i.e., processing capacityequivalent to two CPUs of CPU type CPU-002 run on VMM-001). Minimummemory requirements (not shown) may also be specified by server requestpart 136 in some embodiments. Storage request part 137 includesinformation on requirements for each logical volume requested formigration. In FIG. 12, storage request part 137 has requirements 1261 to1264 for four different volumes, namely a system volume, an indexvolume, a data volume and a log volume. Each request for a volume 1252may include information of requested capacity 1253, access time 1254,read/write ratio 1255 and random/sequential ratio 1256. In otherembodiments, however, some of these requirements may be not specified,or other requirements might be specified such a RAID type, disk type,busy rate, or the like.

Main Flowchart for Finding a Data Center for Migration

FIG. 13 illustrates a flowchart that is representative of the integratedmanagement system processing carried out in an exemplary embodiment ofthe present invention. In this flowchart, there are two data centers, afirst data center is the migration requester and the second data centeris a potential migration acceptor.

At step 1302, migration requester program 123 obtains data regarding theapplication that is to be migrated to another data center. The dataobtained includes data on the network requirements, server requirementsand storage requirements of the application. The requirements for theparticular application may be determined by the migration requesterprogram 123 by determining from VPN resource table 311, virtual machineresource table 411 and storage resource table 511 the resourcescurrently being used by the particular application. Alternatively, therequirements may be supplied with an instruction that instructs themigration requester program 123 to initiate a migration request for theparticular application.

At step 1310, migration requester program 123 in the first data centergenerates a migration request document 13 to attempt to find a suitablecandidate data center to which to migrate an application.

At step 1312, migration requester program 123 retrieves a record fromfederated data center table 213 according to request priority value 1003to select a destination data center to which to send the request formigration, and reads the corresponding value of the request destinationaddress 1002. Alternatively, the migration requester program 123 mightselect all the data centers in the federated data center table, ratherthan proceeding according to the priority values.

At step 1314, migration requester program 123 sends migration requestdocument 13 to the request address 1002 of the destination second datacenter that was read in step 1312.

At step 1320, migration acceptor program 121 of the second data centerreceives the migration request document 13 that was sent from migrationrequester program 123 of the first data center in step 1314, and invokesresource manager program 122 to determine if the migration request canbe accepted.

At step 1322, resource manager program 122 processes the VPN requestpart 135 of migration request document 13, checks the VPN bandwidthcapacity that is available at the second data center in comparison withthe requested bandwidth, and returns the result to migration acceptorprogram 121. The details of the process of checking the VPN capacity areset forth below with reference to FIG. 14.

At step 1324, resource manager program 122 processes the server requestpart 136 of the migration request document 13 by checking availableserver capacity and performance in comparison with the requestedcapacity and performance, and returns the result to migration acceptorprogram 121. The details of the process of checking the server capacityare set forth below with reference to FIG. 15.

At step 1326, resource manager program 122 processes the storage requestpart 137 of the migration request document 13 by checking availablestorage capacity and performance in comparison with that requested, andreturns the results to migration acceptor program 121. The detailedprocess for checking the storage capacity is set forth below withreference to FIG. 16.

At step 1328, migration acceptor program 121 receives the results ofsteps 1322, 1324 and 1326 and determine if the results for all threesteps 1322, 1324, 1326 is “Yes”, i.e., the requested capacity andperformance levels can be satisfied.

At step 1330, if the result for all three steps 1322, 1324, 1326 is“Yes”, then migration can be accepted at the requested levels.

At step 1332, migration acceptor program 121 records the requester,current date and time, and enters the results of steps 1322, 1324 and1326 in network result 1104, server result 1105 and storage result 1106,respectively, in migration request received table 214 of FIG. 11A, andalso enters the contents of the migration request document 13 inmigration request contents 1107. Migration acceptor program 121 thenreturns the results to the migration requester at the first data center.

At step 1340, migration requester program 123 at the first data centerreceives the results from the migration acceptor program at the seconddata center and enters the results in the migration request sent table215 of FIG. 11B, by entering the results of steps 1322, 1324 and 1326 innetwork result 1124, server result 1125 and storage result 1126,respectively.

At step 1342, migration requester program 123 checks whether all threeresults are “Yes” for network result 1124, server result 1125 andstorage result 1126. If so, then a suitable candidate data center hasbeen found for migration of the application and the process skips tostep 1348. On the other hand, if one or more of the results are “No”,the process goes to step 1344 to determine whether there is another datacenter that might meet the requested levels. Further, while the processhas been described as returning only “Yes” or “No” results, the processcould instead return actual capacity values.

At step 1344, if a suitable candidate has not yet been found, migrationrequester program 123 checks whether it has reached the end of therecords listed in federated data center table 213 (i.e., a request hasbeen sent to all other data centers). When the end of the records is notyet reached, migration requester program 123 returns to step 1312 andselects the next data center listed in federated data center table 213.

At step 1346, when no suitable candidate data center has been locatedfrom all the data centers listed in the federated data center table 213,the process may send a request to an administrator inquiring as towhether migration should take place to one of the other data centersanyway. For example, if migration is necessary, an administrator mayinstruct migration to be carried out, even though some requirements ofthe migration request document are not met. For example, a data centerwhich has sufficient storage and processing might be selected to receivemigration even if network bandwidth is insufficient. This step might beparticularly suitable in a case where the results returned show theactual capacity values, rather than just “Yes” or “No” results, thusmaking judgment decisions by the administrator more precise.

At step 1348, migration of the application is carried out by sendingmigration request document 13 to the determined second data center forinstructing migration to take place. The second data center creates newvolumes, copies data to these new volumes, creates new virtual serversor assigns physical servers, configures SAN switch, VLAN and VPN, bootsup these servers, and, along with the first data center, completes anyother necessary steps for carrying out the migration of the application.

Flowchart for Available VPN Bandwidth Confirmation

FIG. 14 illustrates an exemplary detailed flowchart of step 1322 fromthe process of FIG. 13 for checking the VPN free bandwidth with respectto VPN request part 135 of migration request document 13.

At step 1401, resource manager program 122 reads VPN request part 135from migration request document 13.

At step 1402, resource manager program 122 retrieves VPN unallocatedbandwidth from VPN resource table 311 in VPN management server 131, bydetermining the amount of bandwidth 605 that is “Free”.

At step 1403, resource manager program 122 compares the availableunallocated bandwidth with the amount of bandwidth requested in VPNrequest part 135. If the available free bandwidth value is larger thanor equal to the requested bandwidth amount, the process goes to step1404; if not, the process goes to step 1405.

At step 1404, resource manager program 122 sets the result as “YES”.

At step 1405, resource manager program 122 sets the result as “NO”.

At step 1406, resource manager program 122 returns the result from step1404 or 1405 to migration acceptor program 121. For example, in FIG. 6,unallocated bandwidth is 2.0 Mbps. The requested bandwidth in FIG. 12 is1.5 Mbps. So in this example, the result returned by step 1406 is “YES”.Further, while the invention has been described in the environment of aVPN, the invention may also be applied in a non-virtual networkenvironment. For instance, a VPN is one example of network equipmentcapable of providing a specified quality of service to an application.Thus, a feature of the invention includes that the network performanceof applications can be essentially guaranteed prior to determiningwhether to migrate an application. Accordingly, other network equipmentmay be used in place of the VPN, such as by using a suitable mechanismfor providing a predictable available bandwidth, and the like.

Flowchart for Checking Available Server Capacity/Performance

FIG. 15 illustrates an exemplary detailed flowchart of step 1324 of FIG.13 for checking the available server capacity and performance with therespect to server request part 136 of migration request document 13.

At step 1501, resource manager program 122 reads each value on serverrequest part 136, i.e., requested virtual machine manager 1232, CPU type1233 and CPU allocation 1234. For example, according to FIG. 12, theserver request is “VMM-001” for virtual machine manager, “200%” of CPUallocation on CPU type “CPU-002”.

At step 1502, resource manager program 122 searches virtual machineresource table 411 for a record that has the same value as the specifiedVMM and where virtual server column 702 includes a “Free” entry.

At step 1503, if one or more candidate records are found in step 1502,the process goes to step 1504. On the other hand, if no records arelocated that meet the requirements of step 1502, the process goes tostep 1510. For example, according to FIG. 7, table 411 has a record of“VMM-001” for virtual machine manager and an entry of “Free” in acorresponding virtual server column 702, so the result of step 1503would be “Yes” in this example.

At step 1504, resource manager program 122 obtains the value of theperformance capacity 707 for the record located in step 1502. Forexample, according to FIG. 7, the amount of free performance capacity is“185”.

At step 1505, resource manager program 122 searches for a record in CPUprocess time translation table 412 that has the requested virtualmachine manager (i.e., VMM-001 in the example of FIG. 12) and requestedCPU-type (i.e., CPU-002 in the example of FIG. 12), and obtains thecorresponding performance index value 803 for the record. For example,according to FIG. 8, the performance

Index value for “VMM-001 and CPU-002” is “120”.

At step 1506, resource manager program 122 searches for a record in CPUprocess time translation table 412 that has the requested virtualmachine manager and the CPU type located in step 1502, and obtains thecorresponding performance index value 803 from the record. For example,according to FIG. 8, performance index value for “VMM-001 ” and“CPU-001” is “100”.

At step 1507, resource manager program 122 normalizes the requestedperformance index by calculating according to the following formula:(“The value obtained in step 1506”)/(“The value obtained in step1505”)*(“requested CPU allocation”). In the example of FIG. 12 thiscalculation is (100/120)*200=167.

At step 1508, resource manager program 122 compares the normalizedperformance index with the free performance capacity 707 in virtualmachine resource table 411. If the free performance capacity 707 invirtual machine resource table 411 is larger than the normalizedperformance capacity requested, then the process goes to step 1509; ifnot, the process goes to step 1510. In the example, “185”>“167”, so theresult is “Yes”.

At step 1509, resource manager program 122 sets the result as “YES”.

At step 1510, resource manager program 122 sets the result as “NO”.

At step 1511, resource manager program 122 returns the result tomigration acceptor program 121. Thus, it may be seen that the inventionprovides for automatically matching requested performance capacities forservers across different hardware and software platforms. Accordingly, afeature of the invention includes that the server performance ofapplications can be essentially guaranteed prior to determining whetherto migrate a particular application.

Flowchart for Checking Available Storage Capacity/Performance

FIG. 16 illustrates an exemplary detailed flowchart of step 1326 of FIG.13 for checking the available storage capacity and performance withrespect to storage request part 137 of migration request document 13.

At step 1601, resource manager program 122 retrieves one of records1261-1264 from storage request part 137 for processing the requests foreach specified volume individually.

At step 1602, resource manager program 122 reads each value from therecord, i.e., capacity 1253, access time 1254, read/write ratio 1255 andrandom/sequential ratio 1256.

At step 1603, resource manager program 122 searches for an array groupin the data center that meets requested values read in step 1602 forparticular volume. A suitable array group from which to carve the volumemay be located using the following formula:

(Capacity>Requested Capacity)

AND

(Read/Write Ratio*0.9)<Requested Read/Write Ratio<(Read/Write Ratio*1.1)

AND

((Ran./Seq. Ratio*0.9) <Requested Ran./Seq. Ratio<(Ran./Seq. Ratio*1.1))

AND

(Access Time<Requested Access Time)

For instance, according to the example of FIG. 12, the storage requestfor system volume record 1261 specifies a requested capacity 1253 of “10GB”, a requested access time 1254 of “24 ms”, a requested read/writeratio of “80%”, and a requested random/sequential ratio of “10%”.Searching the records in storage volume resource table 511 shows thatrecord 924 meets the requested requirements for the system volume. Theabove formula includes tolerance factors “0.9” and “1.1” which areincluded in the formula to make it easier to find a match for the volumerequest. For example, in some applications, it is not so important thatthe read/write ratio and the random/sequential ratio be a specificnumber, so long as they are within an approximate range. The use ofthese tolerance factors allows this flexibility.

At step 1604, if array group that is able to meet the requestedrequirements for the volume is found, the process goes to step 1605; ifnot, the process goes to step 1606.

At step 1605, resource manager program 122 set the result as “YES”.

At step 1606, resource manager program 122 set the result as “NO”.

At step 1607, resource manager program 122 checks whether the end of allthe records in storage request part 137 has been reached. If all therecords have been processed, the process goes to step 1608; if not, thenthe process goes back to step 1601 to process the next record in thestorage request part 137.

At step 1608, resource manager program 122 returns the results tomigration acceptor program 121. Accordingly, a feature of the inventionincludes that the storage performance of applications can be essentiallyguaranteed prior to determining whether to migrate an application, andthat various different performance metrics or values may be specifiedfor an application, depending on the type and requirements of theparticular application.

Thus, it may be seen that the invention is able to automatically locatean appropriate target data center from among multiple data centers forreceiving migration of an application, while also maintaining thatapplication's performance levels. Additionally, while specificembodiments have been illustrated and described in this specification,those of ordinary skill in the art appreciate that any arrangement thatis calculated to achieve the same purpose may be substituted for thespecific embodiments disclosed. This disclosure is intended to cover anyand all adaptations or variations of the present invention, and it is tobe understood that the above description has been made in anillustrative fashion, and not a restrictive one. Accordingly, the scopeof the invention should properly be determined with reference to theappended claims, along with the full range of equivalents to which suchclaims are entitled.

1. An information system comprising: a plurality of information systems,said information systems each including a network apparatus, one or moreservers, and one or more storage systems, said information systems beingin communication with each other via a network; and a performance indexsetting forth performance values of various types of CPUs operating onvarious server environments; wherein, when migration of an applicationis initiated at a first information system, said first informationsystem is configured to send a migration request from the firstinformation system to one or more targeted information systems of saidplurality of information systems, the migration request specifyingnetwork requirements, server requirements and storage requirements forrunning the application at a first level of performance, wherein each ofthe one or more targeted information systems determines whether thetargeted information system has sufficient free capacity of networkresources, server resources and storage resources to satisfy thespecified network requirements and server requirements for running theapplication at the first level of performance, wherein each of the oneor more targeted information systems sends a response to the firstinformation system including results of whether the targeted informationsystem has said sufficient free capacity, wherein when one or more ofthe targeted information systems has said sufficient free capacity, thefirst information system identifies the one or more targeted informationsystems having said sufficient free capacity as candidate informationsystems for receiving migration of the application, and wherein eachtargeted information system compares the performance value of a firsttype of CPU set forth in the server requirements with the performancevalue of a second type of CPU having available processing capacity atthe targeted information system to determine whether the targetedinformation system has said sufficient free capacity to meet the serverrequirements.
 2. The system according to claim 1, wherein the serverrequirements specify a virtual machine management software type andprocessing capacity of a CPU type, and wherein each targeted informationsystem determines whether the targeted information system has sufficientfree capacity of server resources to satisfy the specified serverrequirements for running the application by determining whether thespecified virtual machine management software type is configured on saidtargeted information system, and whether sufficient processing capacityis available on one or more servers on which said specified virtualmachine is configured to run.
 3. The system according to claim 1,wherein the network requirements included with the migration requestspecify a minimum bandwidth of a virtual private network (VPN), andwherein each targeted information system determines whether sufficientavailable bandwidth exists in the network apparatus at the targetedinformation system for creation of a VPN having the minimum bandwidth ofthe VPN specified in the network requirements.
 4. The system accordingto claim 1, wherein the storage requirements sent with the migrationrequest specify required capacity and attributes of one or more volumesto be created in the target information system, wherein each targetedinformation system determines whether the targeted information systemhas sufficient free capacity of storage resources to satisfy thespecified storage requirements for running the application by carryingout steps of: determining whether an array group at the targetedinformation system has sufficient free storage capacity to satisfy arequested capacity for the one or more volumes; and determining whetherthe array group that has sufficient free storage capacity is able tosatisfy at least one of: a requested read/write ratio, a requestedrandom/sequential ratio, and a requested access time.
 5. The systemaccording to claim 1, further including an integrated management systemat each of said plurality of information systems, wherein saidintegrated management system at each information system is configured tosend and receive migration requests to and from said integratedmanagement systems in other information systems of said plurality ofinformation systems, and wherein said integrated management system ateach information system maintains information on migration requests sentto the other information systems and migration requests received fromthe other information systems.